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IMPROVED VIDEO MOTION PROCESSING 



This invention is directed to picture building in motion compensated video 
processing. 

IVlany contemporary standards conversion and ottier video processing 
systems employ motion compensation in order to Improve the quality of the 
output pictures. In such systems, it is a typical requirement for new output 
pictures to be interpolated from original input pictures. Motion compensation 
assigns motion vectors to the pixels of the input pictures, and these vectors are 
used to project the original pixels to "build" the output picture. 

It is an object of the present Invention to provide techniques for Improving 
the quality of the output pictures of such systems. 

Accordingly, the Invention consists in one aspect in a method of motion 
compensated combination of two pictures of an input picture sequence to form 
an output picture at a temporal location between the two input pictures, 
comprising: projecting input pixels from the input pictures to locations on the 
output picture using motion vectors assigned to those input pixels; counting the 
number of vectors from each input picture which point to a given pixel location on 
the output picture; and employing this count in controlling the mix of the pixels 
projected by those vectors used to produce the output pixel at the given pixel 
location. 

The inventors have thus recognized that counting the number of vector 
"hits" at a particular output pixel location gives important information relating to 
the quality of the eventual output of the motion compensation process. Using this 
count to control the process therefore results in significant advances in quality. 

Preferably, the method comprises employing a non-linear function of the 
count In controlling said mbc. 

In one form of the invention, the method comprises, where a plurality of 
vectors from one of the input pictures point to the given pixel location, assigning 
lower weight to the respective pixels of those vectors from that input picture for 
construction of the pixel at the given location. In another fomn, the method uses 
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an average of the respective pixels of those vectors as the contribution to the 
output pixel from that input picture. 

In still another form, the method comprises, where a plurality of vectors 
point to the given pixel location, taking a median of the vectors, and using the 

5 vector closest to the median for constmction of the output pixel. 

In another aspect, the invention provides a method of motion 
compensated combination of two pictures of an input picture sequence to form 
an output picture at a temporal location between the two input pictures, 
comprising: projecting input pixels from the input pictures to locations on the 

10 output picture using motion vectors assigned to those input pixels; and mixing the 
respective pixels projected by the vectors onto the output picture to produce an 
output pixel at a given location, wherein, where a plurality of vectors from one of 
the input pictures project onto said given pixel location, giving increased 
weighting in controlling the mix to the respective pixels of vectors forming 

15 substantially conjugate pairs. 

The invention will now be described by way of example with reference to 

the accompanying drawings, in which: 

Figures 1 to 3 are diagrams illustrating the function of picture building in a 
20 typical motion compensated system; and 

Figure 4 is a diagram illustrating apparatus according to an embodiment of 
the invention. 

25 • In motion compensated standards conversion , the process of picture 
building is typically important, the accuracy of the process greatly affecting the 
quality of the output images or pictures. The input pictures are typically in the 
form of video fields or frames, though of course, any type of input picture 
sequence may be employed in the embodiments described. IVIotion compensated 

30 picture building techniques are known to the art. and therefore the basic 
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principles will not be discussed in detail here, though some description of the 
problems commonly arising follows. 

In a picture building procedure, as Illustrated in Figure 1, two Input 
pictures, in this case, two video frames (100 and 102) are used to create an 
5 output frame, indicated by dashed line 104. This output frame is to be created at 
a temporal position between the two input frames, though not necessarily 
equidistant from them. 

In order to derive information illustrating the motion occurring between 
input images of an image sequence, a motion measurement process ( of which 
10 the phase correlation technique is preferred) is performed on the input images. 
The resulting motion vectors are assigned to pixels or groups of pixels in the 
input image. 

In the case illustrated in Figure 1 , vectors 106 and 108 have been 
assigned to objects in the two input frames; vector 1 06 points forward 

15 (temporally) towards the output frame position, from a pixel (105) on the first 
input frame (100), and vector 108 points backward from a pixel (107) in the 
second frame. The vectors are used to project the pixels (105, 107) from the 
input frames onto the pixel (1 1 0) of the output frame which is currently being 
constructed. A decision is then taken as to which of the pixels to use, or what 

20 proportion of each pixel to use in a mix of the two. 

The above example, however, is merely a simple case where a single 
vector from each frame may be mapped to the required point. In other cases, 
there may not be a single vector, or there may be multiple vectors pointing to the 
output pixel position . 

25 Figure 2 illustrates one of these cases. Here, a vector (203) projects a 

pixel (202) from the following frame to the output pixel position (204), but there 
are two vectors, 201a and 201b, pointing from different pixels (200a, 200b) on 
the same, previous frame (100), to the same output pixel position (204). This 
may indicate, for example, that one object is moving over another in the current 

30 video sequence. It can be seen that similar situations will arise with multiple 
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vector "hits" from either side of the output position, and with any number of hits 
(greater than one). 

Figure 3 illustrates a different situation. Here, a vector (301) projects a 
pixel (300) from the previous frame to the output pixel position (304), but there is 
5 no vector from the following frame. 

In other cases there may not be a vector pointing to the output point from 
either side, in which case there is simply a hole in the output frame. 

A prior method of picture building, as disclosed in EP 0,648,398, handles 
such situations in the following manner. If there is a single vector "hif from one 

10 frame at the output pixel, the resulting projection of the pixel from that frame is 
assigned a weighting value of 1. If there is a double hit, each vector is given a 
weighting of 1 , giving an overall weighting for that frame or "side" (of the output 
position) of 2. Greater numbers of hits increase the total weighting thus. 
However, if there is no vector hit. the "confidence" in that frame is taken as zero; 

1 5 this therefore prevents the eventual mix of the output pixel taking any information 
firom that frame or side which gave a zero hit result. 

The inventors have recognized that a more sophisticated treatment of 
picture building which measures where multiple and zero hits occur can bring 
significant benefit over this prior technique in the quality of the output pictures. 

20 In embodiments, the invention provides a system which identifies the 

occurrence of such "non-single hits" in the picture building process. The 
techniques described in the following apply the resulting counts to new methods 
of picture building which give the previously unexpected result of greatly 
increasing output picture quality. 

25 In one embodiment, if there is any number of hits, from either of the input 

frames, which is not equal to one, the input from that frame is simply ignored. 
Thus in the case illustrated in Figure 2, the input of both of the pixels 200a and 
200b, projected by vectors 201 a and 201b, would be ignored. The only 
information taken for the output pixel 204 would therefore be that provided by the 

30 following frame (102), from pixel 202 and vector 203. In the case illustrated in 
Figure 3, the number of hits from the following fl-ame is zero (which is not equal 
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to 1) so that frame is ignored, and pixel 300 and vector 301 are used for the 
output pixel (304). 

This method may also be implemented in a "softer" version. For example, 
where a multiple hit occurs, the system may nevertheless Include some 
5 proportion, say 10%, of the offending vectors' source pixels in constructing the 
output pixel. This would be of particular use in cases where there are no hits on 
one side, and multiple hits on the other; at least some of the pixels from those 
vectors which would otherwise be Ignored may be used for the output pixel. 
In most cases, the system will employ some sort of "fallback" mode, in 
1 0 order to prevent failure, or allow a "hole" to appear in the output frame where 
there are no hits from either side. 

Figure 4 Illustrates a detailed version of this embodiment. The previous 
frame is forward projected, and the following frame back projected to an 
intermediate temporal location, and the projections are mixed. This mixing Is 
15 controlled In dependence upon measurements of the number of hits arising on 
either side. Separate "predictions of quality", dependent upon hit count, are 
derived for the previous and following frames, and these are compared to control 
the projection mix. For example. If a single hit is registered for a given pbcel, the 
PoQ is high, whereas If a zero or multiple hit are registered, the PoQ Is low. 
20 Typically, the PoQ will have a step or delta type function where a multiple hit is 
detected, so variation of the PoQ is smoothed over areas having multiple hits, in 
order not to introduce sharp edging effects. 

A separate fallback frame Is also compiled, and a mix between this frame 
and the output of the mixed projected frames is controlled by a prediction of 
25 quality of the projection mix. 

In an alternative embodiment, the median of all vectors pointing to a given 
pixel on the output frame Is taken. A number of options are then available: the 
cicsest vector to the median Is taken, and the oth^r vectors rejected; In a case 
where there is simply a double hit on one side, the offending vector is rejected as 

...+!;er, as the other two vectors are closer to the median; fractions of the 
. j:7Trj5 vectors are taken, according to their proximity to the median. These 
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approaches may be effective in cases where a plurality of spurious vectors 
produce the multiple hits. 

In a further embodiment, the confidence assigned to the vector hits on one 
"side" of the output frame position is normalised. Thus if there is a double hit on 
5 one side, the contribution to the mix may be % of each pixel in the double hit, and 
>2 of the pixel on the other side. 

In a still further embodiment, where there are multiple hits, the vector on 
the "multiple hit side" are compared with those on the other side. If one vector is 
the conjugate (or near conjugate) of one of the vectors on the other side, as in 
10 Figure 2, vectors 201b and 203, then the other vector, 201a, is discarded. 

Essentially, the only vectors taken for the decision on mixing the output pixel are 
such conjugate pairs, as these match the flow of the vector field along the current 
sequence. 

In the embodiments described above, hit counts are generally described 

15 as integer values. In alternatives, if a phase correlation process is implemented 
to sub-pixel accuracy, then a more sophisticated approach is possible. The hit 
count becomes an accumulation over an area of non-integer hit values, rather 
than a simple count of vectors pointing to an integer value. Such "soft" hit counts 
may be processed as in any of the preceding methods in order to provide an 

20 output pixel. 

In general, certain fallback options are required where zero hits or 
spurious vectors occur. For example, if vectors on either side produce an 
inequality or disagreement, the system may take the vector from the closest 
frame to the output temporal position. Where the hit count is zero on both sides, 

25 "holes" occur in the output frame. In such cases, "hole filling" or copying of pixels 
from either frame may be implemented. In other cases, the system may use the 
fallback picture, as in Figure 4. 

In th6 above description of certain embodiments of the invention, the 
example of the projection of two input pictures onto an output picture location Is 

30 used. It should be note.d that aspects of the invention are equally applicable to 
techniques in which more than two input pictures, and their respective pixels and 
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assigned vectors, are used to crieate the output picture. Here, notwithstanding 
the methods described for weighting pixels in particular ways, the proportions of 
pixels used In the final mix may depend to a greater extent upon the distance of 
the Input picture in question from the temporal location of the output picture. 
5 It will be appreciated by those skilled in the art that the invention has been 

described by way of example only, and that a wide variety of alternative 
approaches may be adopted. In particular, the various methods described may 
be used In conjunction. In a variety of advantageous combinations. 



10 
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CLAIMS 

1 . A method of motion compensated combination of a plurality of 
pictures of an input picture sequence to form an output picture at a 
5 temporal location between two of the input pictures, comprising: projecting 

input pixels from the input pictures to locations on the output picture using 
motion vectors assigned to those input pixels; counting the number of 
vectors from each input picture which point to a given pixel location on the 
output picture; and employing this count in controlling the mix of the pixels 
10 projected by those vectors used to produce the output pixel at the given 

pixel location. 



2. A method according to Claim 1, comprising employing a non-linear 
function of the count in controlling said mix. 

15 

3. A method according to Claim 1 or Claim 2, comprising, where a 
plurality of vectors from one of the input pictures point to the given pixel 
location, assigning lower weight to the respective pixels of those vectors 
from that input picture for construction of the pixel at the given location. 

20 

4. A method according to any of the Claims 1 to 3, comprising, where 
a plurality of vectors point to the given pixel location, taking a median of 
the vectors, and using the vector closest to the median for construction of 
the output pixel. 

25 

5. A method according to any of the above claims, comprising, where 
a plurality of vectors from one of the input pictures point to the given pixel 
location, using an average of the respective pixels of those vectors as the 
contribution to the output pixel from that input picture. 
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6, A method of motion compensated combination of a plurality of 
pictures of an input picture sequence to form an output picture at a 
temporal location between two of the input pictures, comprising: projecting 
5 input pixels from the input pictures to locations on the output picture using . 

motion vectors assigned to those input pixels; and mixing the respective 
pixels projected by the vectors onto the output picture to produce an 
output pixel at a given location, wherein, where a plurality of vectors from 
one of the input pictures project onto said given pixel location, giving 
10 Increased weighting in controlling the mix to the respective pixels of 

vectors forming substantially conjugate pairs. 
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